Single-shot high dynamic range imaging

ABSTRACT

A method for generating a high dynamic range image. An image is captured using a Bayer pattern color filter array to generate raw pixel sensor data for the image. The pixel sensor data is separated into highpass (Z i   HP ) and lowpass (Z i   LP ) components. The color components of Z i   HP  are pooled to yield an achromatic highpass data set {circumflex over (X)} i   HP . Saturated pixels in the Z i   LP  components are corrected by borrowing across spectrums to yield the low pass data set {circumflex over (X)} i   LP , and the high dynamic range image is computed as {circumflex over (X)}={circumflex over (X)} i   LP +{circumflex over (X)} i   HP  1. A camera system incorporating this method is also provided.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/528,924, filed on Aug. 30, 2011, entitled “Single-Shot HighDynamic Range Imaging With Conventional Camera Hardware.” The entiredisclosure of the foregoing provisional patent application isincorporated by reference herein.

BACKGROUND

Film and digital photography (including both still and video) generallycannot reproduce the entire available dynamic range range of a scene. Asa result, photographers have always had to find an acceptable balance,trading off highlight details for more shadow details and vice versa.The issue is exacerbated with most digital cameras since the single-chipcolor image sensors used in most digital cameras have a lower dynamicrange than most film.

Recently, high dynamic range (“HDR”) digital imaging has become popular,producing images with considerable tonal range throughout the colorgamut. Typically, such HDR imaging employs two or more captures atvarying exposures (e.g, using exposure bracketing or with the help of aflash). The bracketed exposures are then stitched together (usingpost-capture, post-demosaicking image processing software) to create asingle image having a much greater dynamic tonal range than is possiblein a single image capture with the particular sensor employed. In otherwords, an HDR image is recovered from multiple low dynamic range (“LDR”)images which together comprise a set of overdetermined observationswhich collectively provide an adequate coverage of the dynamic range.Though exposure bracketing is typically implemented as a “timemultiplexing” of exposure rates, capturing moving objects poses aparticular challenge. These time multiplexing techniques also precludeHDR video and handheld camera applications.

Single-shot (i.e., single, non-bracketed exposure) HDR alternatives havebeen proposed as a way to capture HDR images, particularly for HDRimaging of nonstationary objects. For instance, an exposure mosaickingand assorted pixel arrangement has been proposed for implementingspatially varying pixel exposures. Multisensor and alternative pixelarchitecture solutions have also been proposed. However, thesesingle-shot HDR solutions require a special purpose or modifiedhardware.

While a variety of devices and techniques may exist for HDR imaging, itis believed that no one prior to the inventor has made or used aninvention as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

While the specification concludes with claims which particularly pointout and distinctly claim the invention, it is believed the presentinvention will be better understood from the following description ofcertain examples taken in conjunction with the accompanying drawings andimages. In the drawings, like numerals represent like elementsthroughout the several views.

FIG. 1 depicts a schematic illustration of a digital camera whichincludes a processor programmed to perform the method of the presentinvention.

FIG. 2 depicts a series of images reconstructed from RAW pixel sensordata acquired by a conventional DSLR camera, wherein the images ofcolumn (a) are ordinary reconstructions (pixel sensor equalization anddemosaicking) such as typically done in-camera, the images of column (b)were reconstructed from the same RAW data as in column (a) using theS4HDR method described herein (without an additional filter), and theimages of column (c) were reconstructed the S4HDR method describedherein from contemporaneous single shot captures with an 85A filterattached to the lens.

The drawings and images are not intended to be limiting in any way, andit is contemplated that various embodiments of the invention may becarried out in a variety of other ways, including those not necessarilydepicted in the drawings and images. The accompanying drawings andimages incorporated in and forming a part of the specificationillustrate several aspects of the present invention, and together withthe description serve to explain the principles of the invention; itbeing understood, however, that this invention is not limited to theprecise arrangements shown.

DETAILED DESCRIPTION

The following description of certain examples should not be used tolimit the scope of the present invention. Other features, aspects, andadvantages of the versions disclosed herein will become apparent tothose skilled in the art from the following description, which is by wayof illustration, one of the best modes contemplated for carrying out theinvention. As will be realized, the versions described herein arecapable of other different and obvious aspects, all without departingfrom the invention. Accordingly, the drawings and descriptions should beregarded as illustrative in nature and not restrictive.

Unless the context indicates otherwise, the phrase “HDR image” means animage having greater dynamic range than that obtained with the samesensor for a single image using conventional data processing (i.e.,equalization and demosaicking).

Most conventional single-chip color image sensors today make use of acolor filter array (CFA), a spatial multiplexing of absorptive red,green, and blue filters positioned over an array of pixel sensors. Thetypical CFA in most digital cameras employs the Bayer pattern of red,blue and green filters, as described in U.S. Pat. No. 3,971,065 (whichis in incorporated herein by way of reference). The Bayer pattern is arepeating 2×2 arrangement of filters comprising alternating rows ofred-green and green-blue filters. In other words, the first rowcomprises alternating red and green filters (no blue), the second rowcomprises alternating green and blue filters (no red), and this patternrepeats in successive rows. Thus, each 2×2 square in the Bayer patterncontains one red filter, one blue filter and two green filters. Twice asmany green filters are provided as compared to red or blue because thehuman visual system is more responsive to green light.

After an image capture, the raw pixel sensor data (“RAW” format) willgenerally comprise data representing the light intensity measured byeach pixel sensor. However, since each pixel sensor of a Bayer arrayonly measures light intensity of predominantly one color (red, green orblue), the RAW data must be converted into image data representing allthree colors for each pixel. This process is referred to asdemosaicking, and variety of algorithms have been developed for thisprocess. In general, the data from adjacent pixel sensors ismathematically combined in order to provide a three color estimate oflight for each pixel location.

The three color filters in a CFA, however, have different translucency.For example, the green filter in most Bayer pattern CFA's has a greatertranslucency than the red and blue filters, meaning that more photonsare allowed to penetrate through the green filters compared to the redor blue filters. In order to account for these differences, a colorspace transformation matrix must be applied to the RAW pixel sensor datain order to convert that data into a useable color space such as sRGB(standard RGB color space). This conversion essentially equalizes theRAW pixel sensor data by modifying the pixel sensor data to account forthe differences in red, blue and green filter translucency. The matrixfor converting RAW sensor data to a color space such as sRGB ismanufacturer specific, and in many instances camera specific.

By way of example, suppose raw sensor measurements of a GretagMacbeth®ColorChecker® chart are captured under simulated sunlight.Regressing the sample means of the pixel values observed within the samecolored squares of the chart onto published reflectance values producesa matrix Aε

^(3×3) that maps the RAW sensor data for the red, blue and green pixelsensors to the linear sRGB space. For the Nikon® D90™ camera, forexample, the color space transformation matrix is

$\begin{matrix}{A = {\begin{pmatrix}1.001 & {- 0.165} & {- 0.051} \\{- 0.004} & 0.336 & {- 0.095} \\0.034 & {- 0.101} & 0.393\end{pmatrix}.}} & (1)\end{matrix}$The channel “exposure” corresponds to e=A⁻¹ 1ε

³, wherein 1=(1,1,1)^(T) is a neutral linear light in sRGB space. Thus,the channel exposure for the Nikon D90 camera in (1) ise=(1.8271 3.9605 3.4043)^(T)Although A combines the hardware specific (CFA color filtertranslucencies) and the environment (e.g. illuminant) parameters, thecontribution of simulated sunlight to e is limited owing to the relativeflatness of the sunlight spectrum. Hence the differences in translucencyof the red, green, and blue filters of the CFA are largely responsiblefor the unequal diagonal elements of e. From the foregoing, it may alsobe concluded that the translucency of the green CFA filter is more thantwice that of (i.e., more than one stop greater than) the red CFA filterin the Nikon D90 camera, and the translucency of the green filter isonly slightly greater than the blue filter. In practice, channelexposure e is hardware specific. For some manufacturers or cameras, forexample, the translucency of the red and/or blue filters of the CFA maybe higher than that of the green filters of the CFA. However, the colorspace transformation matrix, and hence the channel exposure e may beeasily determined in the same manner described above or is readilyavailable (e.g., from the dcraw open source software).

One reason why the color space transformation matrix and correspondingchannel exposure e is manufacturer, and often times camera specific isthat the relative translucencies of the filters in CFAs varyconsiderably. In fact, some CFAs may exhibit little difference in thetranslucencies of the red, blue and green filters, while in others theremay be even greater differences than that discussed above with respectto the Nikon D90. However, as further discussed herein, even where thefilter translucency differences are small (e.g., less than one stop),one or more photographic filters may be used to magnify (or in somecases create) differences in translucencies between the red, green andblue filters of the CFA.

A major advantage of digital photography is the flexibility ofpostprocessing afforded by the digital format. In particular, colorbalance can be achieved posthoc by performing color space conversion onthe acquired RGB data. A photographic filter mounted on the lens servesa similar purpose for film cameras by changing the spectrum of theincoming light. The spectral response of such photographic filtersmounted on a camera lens are often described in terms of Wratten number.For example, a “warming” filter with a Wratten number of 85A or 85Battenuates the blue channel by ⅔ stops in order to map 5500K colortemperature to 3400K or 3200K, respectively. A “cooling” filter with aWratten number of 80A or 80B has the opposite effect, attenuating thered and green channels by 2 or 1⅔ stops, respectively.

Not surprisingly, the usage of photographic filters for color balancingis rare in digital photography since color balancing can be readilyaccomplished during processing of the raw pixel sensor data. However,these filters do provide a way to magnify (or create) differences in theeffective translucencies of the red, green, and blue CFA filters. Forexample, an 85A or 85B filter on the camera lens will reduce primarilythe number of blue photons (and, to a lesser extent, green photons)striking the sensor array. The end result is equivalent to a reductionin the translucency of the blue filters in the CFA. Mathematically, thechannel exposure is now computed as e=P⁻¹A⁻¹1, wherein P ε

^(3×3) models the attenuation of incoming light that the filterprovides.

By way of example, a Nikon D90 equipped with a Hoya® 85A filterattenuated the red, green, and blue channels by factors of 1.04, 1.51,and 2.70, respectively. Overall, the effective translucency of the greenpixels is at least twice that of the blue pixels, achieving an evengreater leverage for CFA-based exposure bracketing, as further describedherein. It is also possible to stack multiple filters to further amplifythe effects of the filter (e.g., by adding a red absorptive filter inorder to create a greater difference in effective translucency betweenthe green and red pixels). And while the filter is most easilypositioned on the camera lens (e.g., affixed in front of the lens suchas by treaded attachment or the use of a filter mount), one or morefilters can be positioned anywhere along the optical path between thescene being imaged and the pixel sensor array.

In some embodiments of the apparatus and methods herein, the differencein effective translucency between at least two filter colors of the CFAis at least one stop (e.g., red-green and/or green-blue). Thisdifference may be inherent to the color filters of the CFA itself, ormay be provided by the inherent translucencies of the color filters ofthe CFA in combination with one or more color-attenuating filters placedin the optical path (e.g., on the end of the camera lens). In otherembodiments, the difference in effective translucency between at leasttwo pairs of filter colors of the CFA is at least one stop (e.g., onestop difference between red and green, and one stop difference betweenred and blue). In still other embodiments, the difference in effectivetranslucency between all three pairs of filter colors of the RGB Bayerpattern CFA (red-green, green-blue and red-blue) is at least one stop.

In the conventional method of creating HDR images, the key to blendingtogether multiple low dynamic range (“LDR”), exposure-bracketed imagesto form a single HDR image is to draw from optimally exposed regions ofLDR images that are least affected by quantization, noise, andsaturation. For example, when creating a composite HDR image from threecaptures at −1EV, OEV, and +1EV, data from shadow (or low light) regionsof the +1EV dataset will be given greater weight than data from thosesame regions of the −1EV and OEV datasets. Applicant has found that thedifferences in translucency of the red, green and blue CFA filters canbe exploited in a somewhat analogous manner in order to achieve an HDRimage (or an image with increased dynamic range) from a single imagecapture (i.e., single-shot) using conventional camera hardware. Inparticular, HDR (or increased dynamic range) images can be achievedusing conventional CFAs having uniform red, green and blue filters(i.e., every red filter in the CFA is the same, every green filter isthe same, etc.). In addition, although the invention will be describedwith reference to a Bayer CFA, the scope of the present invention is notso limited. For example, the invention may be implemented with CFA'semploying RGBW patterns—red, green, blue and white (transparent to allwavelengths).

Based in part on the notion that red, green, and blue components ofnatural color images are highly correlated, applicant has demonstratedthat the color channels of raw sensor data for a single image capturecomprise a set of overdetermined observations with diverse exposurerates. As further described herein, optimally exposed regions of LDRred/green/blue color components are merged in a principled manner toyield one HDR color image based on rigorous image formation models. Thisapproach will be referred to as spectrally-selective single-shot HDR, or“S4HDR” for short.

Applicant has discovered that the information redundancy in color imagescan be exploited to provide single-shot HDR imaging. One key observationin color image processing that has enabled the likes of demosaicking andcompression is that the spatially highpass components of the red, green,and blue channels are similar. Thus, the color radiance mapX_(i)=(R_(i), G_(i), B_(i))^(T)ε

³ (wherein i=(i_(x), i_(y)) denotes the spatial index of pixels) isseparable into lowpass (^(LP)) and highpass (^(HP)) components:

$\underset{\underset{X_{i}}{︸}}{\begin{pmatrix}R_{i} \\G_{i} \\B_{i}\end{pmatrix}} = {\underset{\underset{X_{i}^{LP}}{︸}}{\begin{pmatrix}R_{i}^{LP} \\G_{i}^{LP} \\B_{i}^{LP}\end{pmatrix}} + {X_{i}^{HP}{\underset{\underset{1}{︸}}{\begin{pmatrix}1 \\1 \\1\end{pmatrix}}.}}}$Here, X_(i) ^(HP) is the highpass shared by all RGB channels. From thisperspective, the lowpass represents the underlying “baseline” thatdescribes textures and edges, and the highpass encodes the “deviation”from the baseline.

Recalling that the channel exposure is e=A⁻¹ 1, and ignoring the spatialsubsampling in the color filter array for the moment, the followingsensor measurement is obtained:

$\begin{matrix}\begin{matrix}{Z_{i} = \left( {r_{i},g_{i},b_{i}} \right)^{T}} \\{= {f\left( {A^{- 1}X_{i}} \right)}} \\{\approx {\max\left( {{\min\left( {{A^{- 1}X_{i}},z_{{ma}\; x}} \right)},z_{m\; i\; n}} \right)}} \\{= {\underset{\underset{Z_{i}^{LP}}{︸}}{\max\left( {{\min\left( {{A^{- 1}X_{i}^{LP}},z_{{ma}\; x}} \right)},z_{m\; i\; n}} \right)} + {\underset{\underset{Z_{i}^{HP}}{︸}}{X_{i}^{HP}\Phi_{i}e}.}}}\end{matrix} & (2)\end{matrix}$In the above equation, ƒ(●) is a monotonic sensor response function,which is assumed to be linear (with slope one) near the middle of thecurve, and saturates when under/over-exposed. Φ_(i) is a diagonal matrixwhose entries indicate saturation (1=not saturated; 0=saturated).Saturated signals are locally lowpass because saturation occurs inbatches of pixels.

Given the relationship of the highpass and lowpass components of sensormeasurement as defined in equation (2) above, HDR image recovery fromtristimulus LDR sensor data Z_(i) can be accomplished. Since X_(i) ^(HP)is overdetermined while X_(i) ^(LP) is potentially underdetermined, eachrequires a different strategy for reconstruction. Applicant's S4HDRmethod generally comprises the following steps:

-   -   1. Sensor data Z_(i) is separated into highpass (Z_(i) ^(HP))        and lowpass (Z_(i) ^(LP)) components using a convolution filter        (e.g., a Gaussian convolution filter).    -   2. Color components of Z_(i) ^(HP) are pooled together to yield        an achromatic highpass data set {circumflex over (X)}_(i) ^(HP)        (see equations (3) and (4) below).    -   3. Saturated pixels in Z_(i) ^(LP) are corrected by borrowing        across spectrums to yield {circumflex over (X)}_(i) ^(LP) (see        equations (6) and (8) below).    -   4. The final HDR image is computed as {circumflex over        (X)}={circumflex over (X)}_(i) ^(LP)+{circumflex over (X)}_(i)        ^(HP) 1.        Theory Behind S4HDR Method

With respect to step 2 of Applicant's S4HDR method, {circumflex over(X)}_(i) ^(HP) may be recovered from Z_(i) ^(HP) based on the followingrelationship:{circumflex over (X)} _(i) ^(HP) :=t _(i) ^(T) Z _(i) ^(HP) =X _(i)^(HP) t _(i) ^(T)Φ_(i) e.  (3)The desired weighting vector t_(i) is an inverse of Φ_(i)e in the sensethat we want t_(i) ^(T)Φ_(i)e i to evaluate to unity. Owing to the factthat the inverse is not unique unless two color components aresaturated, we have the ability to weigh t_(i) by the importance ofindividual color components:(r _(i) ^(HP) ,g _(i) ^(HP) ,b _(i) ^(HP))^(T) =Z _(i) ^(HP)To this end, regions of color components that are better exposed aregiven more weight:

$\begin{matrix}{{t_{i} = \frac{\pi_{i}}{\pi_{i}^{T}\Phi_{i}e}},} & (4)\end{matrix}$where π_(i)=(π_(i) ^(r), π_(i) ^(g), π_(i) ^(b))^(T)ε[0,1]³ is the fuzzymembership of pixels in the nonsaturated region, such as for red:

$\begin{matrix}{\pi_{i}^{r} = {{\frac{z_{{ma}\; x} + z_{m\; i\; n} - {2r_{i}}}{z_{{ma}\; x} + z_{m\; i\; n}}}.}} & (5)\end{matrix}$

The recovery of the lowpass signal X_(i) ^(LP) from Z_(i) ^(LP)—apotentially saturated version of the signal A⁻¹X_(i) ^(LP)—is anunderdetermined problem. Since there are many solutions for X_(i) ^(LP)that map to Z_(i) ^(LP), the solution space of X_(i) ^(LP) must beregularized. One sensible solution among the feasible solutions is the“most neutral” tristimulus value:

$\begin{matrix}{{{{\hat{X}}_{i,{reg}}^{LP} = {{\arg\;{\min\limits_{X_{i}^{LP}}{C_{i}}^{2}}} + {D_{i}}^{2}}},{{{subject}\mspace{14mu}{to}\mspace{14mu} Y_{i}^{LP}} = {\left( {L_{i},C_{i},D_{i}} \right)^{T} = {MX}_{i}^{LP}}}}{{{\Phi_{i}Z_{i}^{LP}} = {\Phi_{i}A^{- 1}M^{- 1}Y_{i}^{LP}}},}} & (6)\end{matrix}$where the matrix M ε

^(3×3) transforms RGB tristimulus values into luminance (L_(i)) andchrominance (C_(i), D_(i)). Linear algebra yields the following closedform solution:{circumflex over (X)} _(i,reg) ^(LP) =AΦ _(i) Z _(i) ^(LP) −A _(s)(A_(s) ^(T) M ^(T) ΨMA _(s))⁻¹ A _(s) ^(T) M ^(T) ΨMAΦ _(i) Z _(i)^(LP)  (7)where Ψ=diag(0,1,1) and A_(s) is a submatrix of A whose rows correspondto the saturated pixels. Intuitively, this is a projection of thenonsaturated pixel components onto a space of feasible colors thatapproximate neutral light. The regularization in (6) is consistent withprior work showing that pixels with high risk for over-exposure likelycorrespond to neutral colors, meaning the grayworld assumption holdsbetter in the region-of-interest. Also, human vision is reportedly lesssensitive to chrominance in the darkest regions of the image (i.e.underexposed pixels). In practice, this scheme succeeds when ∥ΨMAΦ_(i)Z_(i) ^(LP)∥>∥ΨM X_(i) ^(LP)∥ (see equation (7)). To safeguardagainst the possibility that transition towards saturation is a gradualone, the final estimate of X_(i) ^(LP) will be a convex combination:{circumflex over (X)} _(i) ^(LP) =A(diag(π_(i))Z _(i)^(LP)+diag(1−π_(i))A ⁻¹ {circumflex over (X)} _(i,reg) ^(LP)),  (8)where π_(i) is the aforementioned fuzzy membership. The final HDRreconstruction is {circumflex over (X)}={circumflex over (X)}_(i)^(LP)+{circumflex over (X)}_(i) ^(HP)1.Demosaicking Effects on Pixel Sensor Data

Once RAW pixel sensor data has been demosaicked, it is no longerpossible to construct a HDR image from the image. The reason for this isthat demosaicking forces the RGB channels to share the highpass data,hence preventing the use of Applicant's S4HDR method. In particular, theFourier analysis of a Bayer CFA dataset is:

$\begin{matrix}{{F\left\{ \alpha_{i} \right\}(\omega)} + {F\left\{ \beta_{i} \right\}\left( {\omega - \begin{pmatrix}0 \\\pi\end{pmatrix}} \right)} + {F\left\{ \beta_{i} \right\}\left( {\omega - \begin{pmatrix}\pi \\0\end{pmatrix}} \right)} + {F\left\{ \gamma_{i} \right\}\left( {\omega - \begin{pmatrix}\pi \\\pi\end{pmatrix}} \right)}} & (9)\end{matrix}$wherein

{●} is a Fourier tansform operator, ω=(ω_(x), ω_(y)) denote twodimensional spatial frequency, and

$\begin{matrix}{\begin{pmatrix}\alpha_{i} \\\beta_{i} \\\gamma_{i}\end{pmatrix} = {\underset{\underset{N}{︸}}{\begin{pmatrix}\frac{1}{4} & \frac{1}{2} & \frac{1}{4} \\\frac{1}{4} & 0 & {- \frac{1}{4}} \\\frac{1}{4} & {- \frac{1}{2}} & \frac{1}{4}\end{pmatrix}}{Z_{i}.}}} & (10)\end{matrix}$Owing to the invertibility of N, the problem of demosaicking isequivalent to the recovery of (α_(i), β_(i), γ_(i)). The overlappingsupport of the summands in equation (9) indicate aliasing (e.g., both

(α_(i)))(ω) and

(β_(i)))(ω−(_(O) ^(π))) are nonzero for some value of ω); anddemosaicking performance improves when the regions of overlap is reducedby the bandlimitedness of β_(i) and γ_(i). Indeed, the advantage torepresentation in equation (10) is that the difference images β_(i) andγ_(i) enjoy rapid spectral decay (i.e., they do not retain X_(i) ^(HP))and can serve as a proxy for chrominance. On the other hand, the“baseband” image α_(i) can be taken to approximate luminance where X_(i)^(HP) is preserved by the relation:

$\begin{matrix}{{{\hat{X}}_{i}^{HP} \propto \alpha_{i}^{HP}} = {\underset{\underset{weights}{︸}}{\begin{pmatrix}\frac{1}{4} & \frac{1}{2} & \frac{1}{4}\end{pmatrix}}Z_{i}^{HP}}} & (11)\end{matrix}$(from first row of equation (10)). Hence, a demosaicking operating onCFA data is expected to recover the highpass signal {circumflex over(X)}_(i) ^(HP) that is proportional to α_(i) ^(HP) in equation (11).Contrasting the linear combination of Z_(i) ^(HP) in (11) (where theimplied weights are (¼, ½, ¼)) to a more desirable weighting vector of(4) above, it is concluded that (11) ignores the inherent exposurebracketing of a CFA entirely. Thus, the demosaicking output under theordinary scenario is LDR.

Consider an alternative setup. Most modern digital cameras performpost-capture, pre-demosaicking “equalization” aimed at neutralizing theexposure bracketing by scaling red, green, blue channels by the inverseof exposure e. Mathematically, this is equivalent to replacing everyinstance of Z_(i) in (9-11) with diag(e)⁻¹ Z_(i). For example, (10) isupdated as follows:

$\begin{pmatrix}\alpha_{i} \\\beta_{i} \\\gamma_{i}\end{pmatrix} = {{N\;{{diag}(e)}^{- 1}Z_{i}^{LP}} + {X_{i}^{HP}{\underset{\underset{{({1,0,0})}^{T}}{︸}}{N\;{{diag}(e)}^{- 1}e}.}}}$This suggests that equalization improves demosaicking performance—thebandlimitedness assumptions of β_(i) and γ_(i) are robust and the riskof aliasing is reduced. Updating (11) also, the combination ofequalization and demosaicking is expected to recover the highpass signalX_(i) ^(HP) via:

${\hat{X}}_{i}^{HP} = {\alpha_{i}^{HP} = {{\underset{\underset{weights}{︸}}{\begin{pmatrix}\frac{1}{4} & \frac{1}{2} & \frac{1}{4}\end{pmatrix}{{diag}(e)}^{- 1}}Z_{i}^{HP}} = {X_{i}^{HP}.}}}$Comparing this to the desired weighting vector (4), however, the linearcombination of Z_(i) ^(HP) implied by postcapture, pre-demosaickingequalization fails to yield HDR recovery of X_(i) ^(HP).Applicant's Pre-Demosaicking Processing for HDR Image Generation

The key observation of the previous section is that the highpass signal{circumflex over (X)}_(i) ^(HP) recovered from demosaicking isproportional to α_(i) ^(HP). Hence, Applicant has deduced that apost-capture, pre-demosaicking process which precisely controls thelinear combination of Z_(i) ^(HP) in α_(i) ^(HP) will yield a HDR imageupon demosaicking.

Denoting by diagonal matrix W the pre-demosaicking scaling of red,green, and blue channels, equations (9-11) may be updated. For example,demosaicking recovers {circumflex over (X)}_(i) ^(HP) via the relation

$\begin{matrix}{{\hat{X}}_{i}^{HP} = {\alpha_{i}^{HP} = {\underset{\underset{u_{i}^{T}}{︸}}{\begin{pmatrix}\frac{1}{4} & \frac{1}{2} & \frac{1}{4}\end{pmatrix}W}{Z_{i}^{HP}.}}}} & (12)\end{matrix}$The linear weights u_(i) are controlled indirectly by choosing Wintelligently. For example, W may be chosen to satisfy the condition:NWe=(1,0,τ)^(T).  (13)Then, updated equation (11) becomes:

$\begin{matrix}{\begin{pmatrix}\alpha_{i} \\\beta_{i} \\\gamma_{i}\end{pmatrix} = {{NWZ}_{i}^{LP} + {X_{i}^{HP}{\underset{\underset{{({1,0,\tau})}^{T}}{︸}}{NWe}.}}}} & (14)\end{matrix}$Unlike the equalization example in the previous section concerningdemosaicking without preservation of the highpass data, τ is notrequired to be zero—γ_(i) may now have larger support in the frequencydomain as a result. This “relaxation” is justifiable because mostcameras today have exceedingly high spatial resolution compared to whatthe optics can provide. Hence the risk of aliasing between

{α_(i)} and

{γ_(i)} is acceptably low, even though the aliasing risk between

{α_(i)} and

{β_(i)} remains high. Solving for W in (13), the admissible choices of Ware:W _(τ)=diag(e)⁻¹diag((1+τ,1−τ,1+τ)).  (15)

Allowing τ and W_(τ) to be spatially adaptive, the admissible set in(15) which gives more importance to the regions of color components thatare better exposed is chosen. To this effect, one seeks u_(i) that bestapproximates the “ideal weights” in equation (4) in the following sense:

$\begin{matrix}{{\hat{\tau}}_{i} = {\arg\;{\min\limits_{\tau}{{{t_{i}^{T} - {\begin{pmatrix}\frac{1}{4} & \frac{1}{2} & \frac{1}{4}\end{pmatrix}W_{\tau}}}}^{2}.}}}} & (16)\end{matrix}$wherein where t_(i) is as defined previously in equation (3). The closedform solution to this optimization problem is a projection:

${\hat{\tau}}_{i} = {\frac{\left\langle {{t_{i}^{T} - {\left( {\frac{1}{4},\frac{1}{2},\frac{1}{4}} \right){{diag}(e)}^{- 1}}},{\left( {\frac{1}{4},{- \frac{1}{2}},\frac{1}{4}} \right){{diag}(e)}^{- 1}}} \right\rangle}{\left\langle {{\left( {\frac{1}{4},{- \frac{1}{2}},\frac{1}{4}} \right){{diag}(e)}^{- 1}},{\left( {\frac{1}{4},{- \frac{1}{2}},\frac{1}{4}} \right){{diag}(e)}^{- 1}}} \right\rangle}.}$By equation (12), the highpass component of demosaicking output{circumflex over (Z)}_(i) ^(HP) is an HDR reconstruction of X_(i) ^(HP).In other words, by applying a diagonal equalization matrix W (whichtakes into account the different effective translucencies of the filtersof the CFA), the color components of Z_(i) ^(HP) are pooled to yield anachromatic highpass data set {circumflex over (X)}_(i) ^(HP),effectively approximating equations (3) and (4) above. The image dataset is then demosaicked in the usual fashion.

Lastly, the equalization weights W_(i) have no significant effect on therecoverability of X_(i) ^(LP). Therefore, the demosaicking output{circumflex over (Z)}_(i) ^(LP) is processed according to equation (8).The final HDR reconstruction is {circumflex over (X)}={circumflex over(X)}_(i) ^(LP)+{circumflex over (X)}_(i) ^(HP) 1.

In summary, for a system which requires demosaicking, an HDR image maybe generated from the raw pixel sensor data by the method:

(a) capturing an image using a Bayer pattern color filter array togenerate raw pixel sensor data for the image;

(b) apply a weighting matrix W to the pixel sensor data to providemodified pixel sensor data, wherein W gives more importance to theregions of color components that are better exposed;

(c) demosaicking the modified pixel sensor data;

(d) separating the demosaicked modified pixel sensor data into highpass(X^(HP)) and lowpass ({circumflex over (Z)}_(i) ^(HP)) components;

(e) correcting saturated pixels in the {circumflex over (Z)}_(i) ^(LP)components by borrowing across spectrums to yield the low pass data set{circumflex over (X)}_(i) ^(LP); and

(e) computing the high dynamic range image {circumflex over(X)}={circumflex over (X)}_(i) ^(LP)+{circumflex over (X)}_(i) ^(HP) 1.

Because the methods and systems described herein can be accomplishedusing existing hardware, particularly conventional color filter arrays(CFAs) and their associated image sensor arrays, the single-shot HDRimaging described herein may be incorporated into existing still andvideo cameras (including cell phone cameras) by modifying (also referredto as updating) the firmware or other program instructions used toprocess sensor data in the imaging device. In this manner, cameras witha limited bit depth datapath will provide the performance of camerashaving more complex hardware. These methods and systems are particularlyuseful for vehicular vision systems which are typically called on tocapture images (still or video) in rapidly changing environments (e.g.,rapidly changing light) as well as dynamic scenes. Although thesingle-shot HDR systems and methods described herein generally will notyield results superior to those achieved using time multiplexing HDRimaging, none of these benefits are available to the existingsingle-shot or multishot HDR solutions.

These differences become even more noticeable when an additional filterdesigned to attenuate certain spectra of incoming light is applied tothe optical path of a camera. Whether these properties of CFA andphotographic filters can be exploited for the purpose of high dynamicrange (HDR) imaging is a question that has received surprisingly littleattention in the extant literature.

In some embodiments, the sensor data is processed in-camera, while inother embodiments the sensor data is processed outside of the camerausing image processing software. FIG. 1 is a schematic illustration of acamera system 200 which incorporates the methods of the presentinvention. Various embodiments of the present invention may include oneor more cameras or other image capturing devices. Camera system 200includes a plurality of light sensitive elements 201 (pixel sensorelements). Each light sensitive element is configured to measure amagnitude of light 203 corresponding to a location within an image beingcaptured. Light 203 passes through the lens, which is schematicallyshown in FIG. 1 to have a filter 206 mounted on the front of the lens inorder to alter the effective translucency of one or more filters of theCFA 205 (e.g., an 80A, 80B, 85A or 85B). Of course more than one suchfilter 206 may be attached to the front of the lens, as describedpreviously herein.

The plurality of light sensitive elements 201 may include a plurality ofphoto sensitive capacitors of a charge-coupled device (CCD).Alternatively, the plurality of light sensitive elements 201 may includeone or more complementary metal-oxide-semiconductor (CMOS). During imagecapture, each photosensitive capacitor may be exposed to light 203 for adesired period of time, thereby generating an electric chargeproportional to a magnitude of the light at a corresponding imagelocation. After the desired period of time, the electric charges of eachof the photosensitive capacitors may then be measured to determine thecorresponding magnitudes of light at each image location in order togenerate RAW pixel data for the image.

A Bayer pattern CFA 205 is positioned over the pixel sensor array 201may be disposed on one or more of the light sensitive elements 201. Anindication of the magnitudes of light measured by each light sensitiveelement is transmitted to at least one processor 209. In one embodimentin which a plurality of photosensitive capacitors of a CCD are used aslight sensitive elements 201, the current in each photosensitivecapacitor is measured and converted into a signal that is transmittedfrom the CCD to the processor 209. In some embodiments, the processor209 includes a general purpose microprocessor and/or an applicationspecific integrated circuit (ASIC) and/or a field programmable gatearray(s). In some embodiments, the processor includes memory elements(e.g., registers, RAM, ROM) configured to store data (e.g., measuredmagnitudes of light, processing instructions, demosaiced representationsof the original image). In some embodiments, the processor 209 is partof the image capturing device (e.g., camera system 200). In otherembodiments, the processor 209 is part of a general purpose computer orother computing device.

In some embodiments, the processor 209 is coupled to a communicationnetwork 211 (e.g., a bus, the Internet, a LAN). In some embodiments, oneor more storage components 213, a display component 215, a networkinterface component (not shown), a user interface component 217, and/orany other desired component are coupled to the communication network 211and communicate with the processor 209. In some implementations, thestorage components 213 include nonvolatile storage components (e.g.,memory cards, hard drives, ROM) and/or volatile memory (e.g., RAM). Insome implementations, the storage components 213 are used to storemosaiced and/or demosaiced representations of images captured using thelight sensitive elements 201.

Processor 209 is configured to perform a plurality of processingfunctions, such as responding to user input, processing image data fromthe photosensitive elements 201, and/or controlling the storage anddisplay elements 213, 215. In particular, one or more such processors209 are configured to perform the image data processing functionsdescribed above.

In some embodiments, the image capturing device 200 comprises a videocamera configured to capture representations of a series of images. Inaddition to or as an alternative to capturing a representation of asingle image, as described above, such a video camera may capture aplurality of representations of a plurality of images over time. Theplurality of representations may comprise a video. The video may bestored on a machine readable medium in any format, such as a MPEG or anyother electronic file format.

While several devices and components thereof have been discussed indetail above, it should be understood that the components, features,configurations, and methods of using the devices discussed are notlimited to the contexts provided above. In particular, components,features, configurations, and methods of use described in the context ofone of the devices may be incorporated into any of the other devices.Furthermore, not limited to the further description provided below,additional and alternative suitable components, features,configurations, and methods of using the devices, as well as variousways in which the teachings herein may be combined and interchanged,will be apparent to those of ordinary skill in the art in view of theteachings herein.

Having shown and described various versions in the present disclosure,further adaptations of the methods and systems described herein may beaccomplished by appropriate modifications by one of ordinary skill inthe art without departing from the scope of the present invention.Several of such potential modifications have been mentioned, and otherswill be apparent to those skilled in the art. For instance, theexamples, versions, geometrics, materials, dimensions, ratios, steps,and the like discussed above are illustrative and are not required.Accordingly, the scope of the present invention should be considered interms of the following claims and is understood not to be limited to thedetails of structure and operation shown and described in thespecification and drawings.

What is claimed is:
 1. A method for generating a high dynamic rangeimage from a single image capture, comprising: (a) capturing a singleimage at a single exposure using a Bayer pattern color filter array togenerate raw pixel sensor data for the single image; (b) separating theraw pixel sensor data into highpass (Z_(i) ^(HP)) and lowpass (Z_(i)^(LP)) components; (c) pooling together the color components of Z_(i)^(HP) to yield an achromatic highpass data set {circumflex over (X)}_(i)^(HP); (d) correcting saturated pixels in the Z_(i) ^(LP) components byborrowing across spectrums to yield the low pass data set {circumflexover (X)}_(i) ^(LP); and (e) computing the high dynamic range image{circumflex over (X)}_(i)={circumflex over (X)}_(i) ^(LP)+{circumflexover (X)}_(i) ^(HP) (1,1,1)^(T), where (1,1,1)^(T) is neutral light. 2.A camera system configured to capture and process image data inaccordance with the method of claim
 1. 3. A method for generating a highdynamic range image from a single image capture, comprising: (a)capturing a single image at a single exposure using a Bayer patterncolor filter array to generate raw pixel sensor data for the singleimage; (b) applying a weighting matrix W to the pixel sensor data toprovide modified pixel sensor data, wherein W gives more importance tothe regions of color components that are better exposed; (c)demosaicking the modified pixel sensor data; (d) separating thedemosaicked modified pixel sensor data into highpass (X^(HP)) andlowpass ({circumflex over (Z)}_(i) ^(LP)) components; (e) correctingsaturated pixels in the {circumflex over (Z)}_(i) ^(LP) components byborrowing across spectrums to yield the low pass data set {circumflexover (X)}_(i) ^(LP); and (f) computing the high dynamic range image{circumflex over (X)}={circumflex over (X)}_(i) ^(LP)+{circumflex over(X)}_(i) ^(HP) (1,1,1)^(T) where (1,1,1)^(T) is neutral light.
 4. Acamera system configured to capture and process image data in accordancewith the method of claim
 3. 5. The method of claim 3 wherein theweighting matrix is chosen from the set:W _(τ)=diag(e)⁻¹diag((1+τ,1−τ,1+τ)).
 6. A method for generating a highdynamic range image from a single image capture, comprising: (a)capturing a single image at a single exposure by a pixel sensor arrayhaving a color filter array, thereby generating raw pixel sensor datafor the single image capture, the color filter array comprising an arrayof red, blue and green filters; (b) applying a diagonal equalizationweighting matrix W to the pixel sensor data to provide modified pixelsensor data comprising an achromatic highpass data set {circumflex over(X)}_(i) ^(HP), wherein W gives more importance to the regions of colorcomponents that are better exposed; (c) correcting saturated pixels inthe lowpass ({circumflex over (Z)}_(i) ^(LP)) components by borrowingacross spectrums to yield the low pass data set {circumflex over(X)}_(i) ^(LP); and (d) computing the high dynamic range image{circumflex over (X)}={circumflex over (X)}_(i) ^(LP)+{circumflex over(X)}_(i) ^(HP) (1,1,1)^(T) where (1,1,1)^(T) is neutral light.
 7. Themethod of claim 6, wherein said color filter array consists of an arrayof identical red filters, identical green filters, and identical bluefilters.
 8. The method of claim 6, wherein said color filter arrayconsists of an RGBW array.
 9. The method of claim 6, further comprisingthe step of demosaicking the modified pixel sensor data after said stepof applying a diagonal equalization weighting matrix, and before saidstep of correcting saturated pixels in the lowpass ({circumflex over(Z)}_(i) ^(LP)) components.
 10. A camera system configured to captureand process image data in accordance with the method of claim
 9. 11. Themethod of claim 6, wherein the weighting matrix is chosen from the set:W _(τ)=diag(e)⁻¹diag((1+τ,1−τ,1+τ)).
 12. The method of claim 6, whereinthe difference in effective translucency between at least two filtercolors of the color filter array is at least one stop.
 13. The method ofclaim 12, wherein the difference in effective translucency between allthree pairs of filter colors of the color filter array is at least onestop.
 14. The method of claim 6, wherein one or more additional filtersare positioned within an optical path between the scene being capturedand the pixel sensor array in order to alter the effective translucencyof one or more filter colors in the color filter array.
 15. The methodof claim 6, wherein the image is captured by a camera, and steps (b)-(d)are performed in a processor outside of the camera using imageprocessing software.
 16. A camera system configured to capture andprocess image data in accordance with the method of claim
 6. 17. A videocamera system configured to capture and process image data in accordancewith the method of claim 6 for generating video comprising a pluralityof high dynamic range images.
 18. An image processing system forgenerating a high dynamic range digital image from raw pixel sensor datarepresenting the light intensity measured by each pixel sensor of animage capturing device of a single image capture, wherein the imagecapturing device is adapted to capture images using a color filter arrayhaving a plurality of uniform red, green and blue filters, the systemcomprising a processor configured to perform the steps of: (a) apply adiagonal equalization weighting matrix W to the pixel sensor data toprovide modified pixel sensor data comprising an achromatic highpassdata set {circumflex over (X)}_(i) ^(HP), wherein W gives moreimportance to the regions of color components that are better exposed;(b) correct saturated pixels in the lowpass {circumflex over (Z)}_(i)^(LP) components by borrowing across spectrums to yield the low passdata set {circumflex over (X)}_(i) ^(LP); and (c) compute the highdynamic range image {circumflex over (X)}={circumflex over (X)}_(i)^(LP)+{circumflex over (X)}_(i) ^(HP) (1,1,1)^(T) where (1,1,1)^(T) isneutral light.